Method and Apparatus of Correcting Hybrid Flash Artifacts in Digital Images

ABSTRACT

A method for digital image eye artifact detection and correction include identifying one or more candidate red-eye defect regions in an acquired image. For one or more candidate red-eye regions, a seed pixels and/or a region of pixels having a high intensity value in the vicinity of the candidate red-eye region is identified. The shape, roundness or other eye-related characteristic of a combined hybrid region including the candidate red-eye region and the region of high intensity pixels is analyzed. Based on the analysis of the eye-related characteristic of the combined hybrid region, it is determined whether to apply flash artifact correction, including red eye correction of the candidate red-eye region and/or correction of the region of high intensity pixels.

CROSS-REFERENCE TO RELATED APPLICATIONS

This application is a Continuation of U.S. patent application Ser. No.13/357,548, filed Jan. 24, 2012; which is a Continuation of U.S. patentapplication Ser. No. 12/960,487, filed Dec. 4, 2010, now U.S. Pat. No.8,126,265; which is a Continuation of U.S. patent application Ser. No.12/558,859, filed on Sep. 14, 2009, now U.S. Pat. No. 7,865,036; whichis a Continuation of U.S. patent application Ser. No. 11/282,955, filedon Nov. 18, 2005, now U.S. Pat. No. 7,599,577, entitled, “Method andApparatus of Correcting Hybrid Flash Artifacts in Digital Images.”

BACKGROUND

1. Field of the Invention

The present invention relates to digital image correction, andparticularly to correction of eye artifacts due to flash exposure.

2. Description of the Related Art

U.S. Pat. No. 6,873,743 to Steinberg, which is hereby incorporated byreference, discloses an automatic, red-eye detection and correctionsystem for digital images including a red-eye detector module thatdetermines without user intervention if a red-eye defect exists. If adefect is located in an image the portion of the image surrounding thedefect is passed to a correction module that de-saturates the redcomponents of the defect while preserving the other colorcharacteristics of the defect region.

WO03/071484, Pixology, discloses a method of detecting red-eye featuresin a digital image comprising identifying highlight i.e. glint regionsof the image having pixels with a substantially red hue and highersaturation and lightness values than pixels in the regions therearound.In addition, pupil regions comprising two saturation peaks either sideof a saturation trough may be identified. It is then determined whethereach highlight or pupil region corresponds to part of a red-eye featureon the basis of further selection criteria, which may includedetermining whether there is an isolated, substantially circular area ofcorrectable pixels around a reference pixel. Correction of red-eyefeatures involves reducing the lightness and/or saturation of some orall of the pixels in the red-eye feature.

In many cases, the eye-artifact that is caused by the use of flash ismore complex than a mere combination of red color and a highlight glint.Such artifacts can take the form of a complex pattern of hybrid portionsthat are red and other portions that are yellow, golden, white or acombination thereof. One example includes the case when the subject doesnot look directly at the camera when a flash photograph is taken. Lightfrom the flash hits the eye-ball at an angle which may provokereflections different than retro-reflection, that are white or goldencolor. Other cases include subjects that may be wearing contact lensesor subjects wearing eye glasses that diffract some portions of the lightdifferently than others. In addition, the location of the flash relativeto the lens, e.g. under the lens, may exacerbate a split discolorationof the eyes.

SUMMARY OF THE INVENTION

A technique is provided for digital image artifact correction asfollows. A digital image is acquired. A candidate red-eye defect regionis identified in the image. A region of high intensity pixels isidentified which has at least a threshold intensity value in a vicinityof said candidate red-eye region. An eye-related characteristic of acombined hybrid region is analyzed. The combined hybrid region includesthe candidate red-eye region and the region of high intensity pixels.The combined hybrid region is identified as a flash artifact regionbased on the analyzing of the eye-related characteristic. Flash artifactcorrection is applied to the flash artifact region.

The flash artifact correction may include red-eye correction of thecandidate red-eye region. The flash artifact correction may also includecorrection of the region of high intensity pixels.

A bounding box may be defined around the candidate red-eye defectregion. The identifying of the region of high intensity pixels maycomprise identifying a seed high intensity pixel by locating said seedhigh intensity pixel within said bounding box. The seed pixel may have ayellowness above a pre-determined threshold and a redness below apre-determined threshold. The region of high intensity pixels may bedefined around the seed pixel.

The analyzing may include calculating a difference in roundness betweenthe candidate red-eye region and the combined region. The red-eyecorrection may be applied when the roundness of the combined hybridregion is greater than a threshold value.

The method may include determining to apply red-eye correction when aroundness of the combined hybrid region is greater than a roundness ofthe candidate red-eye region by a threshold amount.

The method may include determining to not apply correction when theregion of high intensity pixels includes greater than a threshold area.The area may be determined as a relative function to the size of saidbounding box.

The method may include determining a yellowness and a non-pinkness ofthe region of high intensity pixels. The acquired image may be in LABcolor space, and the method may include measuring an average b value ofthe region of high intensity pixels and determining a difference betweenan average a value and the average b value of the region of highintensity pixels.

The analyzing may include analyzing the combined hybrid region for thepresence of a glint, and responsive to detecting a glint, determining tonot correct the region of high intensity pixels responsive to thepresence of glint.

The method may include correcting the region of high intensity pixels byselecting one or more pixel values from a corrected red-eye region andemploying the pixel values to correct the region of high intensitypixels. The selected pixel values may be taken from pixels having L andb values falling within a median for the corrected red-eye region.

The method may include determining to not apply correction when anaverage b value of the region of high intensity pixels exceeds arelatively low threshold or if a difference between average a and bvalues is lower than a pre-determined threshold.

The method may include converting the acquired image to one of RGB, YCCor Lab color space formats, or combinations thereof.

The analyzing of the acquired image may be performed in Luminancechrominance color space and the region of high intensity pixels may havea luminance value greater than a luminance threshold, and blue-yellowchrominance values greater than a chrominance threshold and a red-greenvalue less than a red-green threshold.

The method may include filtering the red-eye candidate regions toconfirm or reject said regions as red-eye defect regions, and selectinga subset of the rejected red-eye candidate regions.

The method may be implemented within a digital image acquisition device.The method may be implemented as part of an image acquisition process.The method may be implemented as part of a playback option in thedigital image acquisition device.

The method may be implemented to run as a background process in adigital image acquisition device. The method may be implemented within ageneral purpose computing device and wherein the acquiring may includereceiving the digital image from a digital image acquisition device.

The candidate red-eye region and/or the region of high intensity pixelsmay be corrected. The region of high intensity pixels may be correctedafter the red-eye candidate region. The correcting of the region of highintensity pixels may utilize corrected pixel values based on thecandidate red-eye region. Results of correcting the candidate red-eyeregion and the region of high intensity pixels may be combined in such amanner as to obfuscate a seam between the regions. The method mayinclude smoothing a seam region between the candidate red-eye region andthe region of high intensity pixels.

The eye-related characteristic may include shape, roundness, and/orrelative pupil size.

A further method is provided for digital image artifact correction. Adigital image is acquired. A candidate red-eye defect region isidentified in the image. A seed pixel is identified which has a highintensity value in the vicinity of the candidate red-eye region. Aneye-related characteristic of a combined hybrid region is analyzed. Thecombined hybrid region includes the candidate red-eye region and theseed pixel. The combined hybrid region is identified as a flash artifactregion based on the analyzing of the eye-related characteristic. Flashartifact correction is applied to the flash artifact region.

The flash artifact correction may include red-eye correction of thecandidate red-eye region. The flash artifact correction may also includecorrection of a second region that includes the seed pixel.

The seed pixel may have a yellowness above a pre-determined thresholdand a redness below a pre-determined threshold.

The method may include filtering the red-eye candidate regions toconfirm or reject the regions as red-eye defect regions, and selecting asubset of the rejected red-eye candidate regions.

The method may be implemented within a digital image acquisition device.The method may be implemented as part of an image acquisition process.The method may be implemented as part of a playback option in thedigital image acquisition device.

The method may be implemented to run as a background process in adigital image acquisition device. The method may be implemented within ageneral purpose computing device, and the acquiring may includereceiving the digital image from a digital image acquisition device. Theanalyzing may include checking whether an average b value exceeds arelatively low threshold. The analyzing may include checking whether adifference between an average a value and the average b value is lowerthan a given threshold.

BRIEF DESCRIPTION OF THE DRAWINGS

Embodiments of the invention will now be described by way of examplewith reference to the accompanying drawings, in which:

FIG. 1 illustrates an image in which several defect candidate regionshave been identified and surrounded by bounding boxes;

FIG. 2 shows in more detail a candidate region exhibiting a half-redhalf-white/golden defect; and

FIG. 3 illustrates a flow diagram of an embodiment of image correctionsoftware according to the present invention.

DETAILED DESCRIPTION OF THE PREFERRED EMBODIMENTS

The preferred embodiments provide improved methods for detecting defectsin subjects' eyes as well as methods for correcting such defects.

A preferred embodiment may operate by examining a candidate red eyeregion, looking in its neighborhood or vicinity for a possible yellow,white and/or golden patch belonging to the same eye, and, if any, undercertain conditions correcting one or both of the red-eye or goldenpatch.

Using a technique in accordance with a preferred embodiment, the qualityand acceptability of automatic eye correction can be increased for halfred—half white/golden defects.

Implementations of the preferred embodiments can take advantage of thered part of the eye defect being detected by one automatic red-eyedetection processing method, perhaps utilizing a conventional techniqueor a new technique, so the detection of the non-red regions can beapplied as a pre-correction stage, and so that this method may take fulladvantage of existing or new detection methods. The correction parts ofsuch red-eye processing may be altered to implement a technique inaccordance with a preferred embodiment, while non correction partspreferably are not altered.

A technique in accordance with a preferred embodiment may provide aqualitative improvement in image correction with relatively littleprocessing overhead making it readily implemented in cameras that mayhave limited processing capability and/or without unduly effecting thecamera click-to-click interval.

It will be seen that pixels belonging to a red-eye defect may becorrected by reducing the red value of the pixel. As an example, imageinformation may be available in Luminance-Chrominance space such asL*a*b* color space. This may involve reducing the L* and a* value of apixel to a suitable level. In many cases, reduction of the a* value mayautomatically restore the chrominance of the eye thus restoring a truevalue of the iris.

However, for white/golden pixels of a half red—half white/golden eyedefect, the L and possibly b characteristics of the pixel may also beeither saturated and/or distorted. This means that unlike red eyedefects, in these cases the original image information may be partiallyor even totally lost. The correction may be performed by reducing theoverall L* value as well as reduction of the a* and b*. However, because1* may be very high, the chrominance may be very low, thus there may notbe significant color information remaining. In an additional preferredembodiment, correction of the white/golden portion of the defectinvolves reconstructing the eye, as opposed to the restoration describedabove from information from the corrected red eye portion of the defect.

Referring now to FIG. 3, a digital image 10 may be acquired 30 in anotherwise conventional manner and/or utilizing some innovativetechnique. Where the embodiment is implemented in a device separate froma device such as a camera or scanner on which the image was originallyacquired, the image may be acquired through file transfer by anothersuitable means including wired or wireless peer-to-peer or networktransfer. Otherwise the image correction process described below, ifsuitably speed optimized, can either be implemented within the imageacquisition chain of the image acquisition device for displaying acorrected image to a user before the user chooses to save and/or acquirea subsequent image; or alternatively, the image correction process canbe analysis optimized to operate in the background on the imageacquisition device on images which have been stored previously.

Next, during red-eye detection 32, red-pixels 20 are identified andsubsequently grouped into regions 22 comprising a plurality ofcontiguous (or generally contiguous) pixels (see, e.g., FIG. 2). Theseregions can be associated 34 with larger bounding box regions12,14,16,18 (see, e.g., FIG. 1). The candidate regions contained withinthese bounding boxes are then passed through a set of filters 36 todetermine whether the regions are in fact red-eye defects or not.Examples of such falsing filters are disclosed in U.S. Pat. No.6,873,743.

One possible reason a filtering process might reject a candidate region,such as a region of red-pixels 20 as illustrated at FIG. 2, is that itlacks the roundness expected of a typical red-eye defect. Such regionsas well as regions failed for other suitable reasons may be preferablypassed as rejected regions 38 for further processing to determine ifthey include a half red—half white/golden eye defect—and if so for thedefect to be corrected accordingly. Much of the operation of thisprocessing can be performed in parallel with other red-eye processing(in for example a multi-processing environment) or indeed processing foreach rejected region could be carried out to some extent in parallel.

Processing in accordance with an exemplary embodiment which may beinvolved in checking for half red—half white/golden eye defects isoutlined in more detail as follows:

1. The bounding box 12-18 of an already detected red part of the eyeartifact is searched 40 for a point, say 26 (see FIG. 2) having:

a. High intensity (I>threshold)

b. High yellowness (b>threshold)

c. Low redness (a<threshold)

In this example, it is assumed that the image information for a regionis available in Lab color space, although another embodiment couldequally be implemented for image information in other formats such asRGB, YCC or indeed bitmap format.

If such a point does not exist, then STOP (i.e., the decision is takenthat no white/golden patch exists in the vicinity of the red area) andconfirm that the region is to be rejected 42.

2. Starting from a point detected in Step 40, grow 44 a region 24 (seeFIG. 2) based on luminance information, for example, if luminance isgreater than a threshold, a point is added to the white/golden region24. If the region 24 exceeds a predefined maximum allowable size, step46, then STOP and confirm that the region is to be rejected 42. Themaximum allowable size can be determined from a ratio of the boundingbox area vis-à-vis the overall area of the red 22 and white/goldenregion 24.

3. Yellowness and non-pinkness of the white region are then assessed 48by checking that average b value exceeds a relatively low threshold, andthe difference between average “a” and average “b” is lower than a giventhreshold. If at least one test fails, then STOP and confirm that theregion is to be rejected 42.

4. In this embodiment, the increase of roundness of the combination ofinitial red 22 and detected white/golden regions 24 from the originalred region 22 is checked 50. Thus, the roundness of the union of the redand white/golden regions is computed and compared with that of the redregion 22. If roundness is less than a threshold value or decreased ornot increased sufficiently by “adding” the white/golden region 24 to thered one 22, then STOP and reject the region 42. Roundness of a region ispreferably computed using the formula

${Roundness} = \frac{{Perimeter}^{2}}{4{\pi \cdot {Area}}}$

Prior to assessing roundness, a hole filling procedure is preferablyapplied to each region 22,24 to include for example pixel 28 within theunion.

5. If the region passes one or more and preferably all of the abovetests, it is added to the list of confirmed red-eye regions. At thispoint, the red part of the eye defect can be corrected 52 in any ofvarious manners, for example, by reducing the a value of pixels in Labcolor space, while the pixels that were corrected are marked to be usedin further processing.

6. For white/golden regions that were added to the list of red-eyedefect regions, further correction of the white/golden portion of thedefect can be applied, after some further checks. One such check is todetect glint 54. In RGB space, glint candidates are selected as highluminance pixels (for example, min(R, G)>=220 and max(R, G)==255). If avery round (e.g., in one or both of aspect ratio and elongation),luminous, and desaturated region is found within the interior of thecurrent “red ∪ white” region 22,24, its pixels may be removed from the“pixels-to-correct” list. The glint may be the entire high luminanceregion but in most cases only a small part of the high luminance regionwill satisfy the criteria for glint pixels.

7. Where a glint is not detected or is small relative to the size of thewhite/golden region, the non-red eye artifact pixels 24 can be corrected56 preferably taking color information from red pixels 22 which wherealready corrected at step 52, if such information after the correctionexists. Alternatively, the correction can be done by reduction of theLuminance value. In the preferred embodiment, color information isderived from a selection of ex-red pixels with L and b values which liein the median for that region (between the 30% and 70% points on acumulative histogram for L and b). These color samples (from the alreadycorrected red part of the eye) are used to create the same texture onboth the red and non-red defect parts of the eye. It should be notedthat the L and b histograms may be generally available frompreprocessing steps, for example, those for determining variousthresholds, and won't necessarily have changed during correction as thered correction may just involve reducing the a value of a pixel. It ispossible that the correction of the red-eye region and the one for thehigh intensity region may show an unpleasant seam between the regions.In an alternative embodiment, the corrected region will be smoothed insuch a manner that the seams between the two regions if exist, will beeliminated.

The present invention is not limited to the embodiments described aboveherein, which may be amended or modified without departing from thescope of the present invention as set forth in the appended claims, andstructural and functional equivalents thereof.

In methods that may be performed according to preferred embodimentsherein and that may have been described above and/or claimed below, theoperations have been described in selected typographical sequences.However, the sequences have been selected and so ordered fortypographical convenience and are not intended to imply any particularorder for performing the operations.

In addition, all references cited above herein, in addition to thebackground and summary of the invention sections, are herebyincorporated by reference into the detailed description of the preferredembodiments as disclosing alternative embodiments and components.

1. A digital image acquisition device comprising: a lens; an imagesensor; a processor; and a computer readable medium having computerreadable code embodied therein for programming the processor to analyzea digital image for segmenting an iris region from one or more adjacentsub-regions of an identified eye region within the image, wherein thedevice is configured to: acquire a digital image; identify a candidateeye region in said image; determine one or more sub-regions of thecandidate eye region each having an intensity value that is above orbelow one or more threshold intensity values; identify an iris regionand one or more adjacent sub-regions based on discerning chrominancevalues of pixels of the iris region and the one or more adjacentsub-regions; and determine the iris region based on analyzing one ormore chrominance values of pixels of at least one of the adjacentsub-regions.
 2. The device of claim 1, the device being furtherconfigured to analyze a shape characteristic of the one or moresub-regions.
 3. The device of claim 1, wherein said device is furtherconfigured to analyze said acquired image in Luminance chrominance colorspace and said image comprises a region of high intensity pixels havinga luminance value greater than a luminance threshold.
 4. The device ofclaim 3, wherein said region of high intensity pixels has blue-yellowchrominance values greater than a chrominance threshold.
 5. The deviceof claim 4, wherein said region of high intensity pixels has a red-greenvalue less than a red-green threshold.
 6. The device of claim 1, whereinsaid device is further configured to analyze an eye-relatedcharacteristic comprising at least one of shape, roundness and pupilsize.
 7. A non-transitory computer readable medium having computerreadable code embodied therein for programming one or more processors toanalyze an acquired digital image for segmenting an iris region from oneor more adjacent sub-regions of an identified eye region within theimage, wherein the code is configured to program the one or moreprocessors to: identify a candidate eye region in said image; determineone or more sub-regions of the candidate eye region each having anintensity value that is above or below one or more threshold intensityvalues; identify an iris region and one or more adjacent sub-regionsbased on discerning chrominance values of pixels of the iris region andthe one or more adjacent sub-regions; and determine the iris regionbased on analyzing one or more chrominance values of pixels of at leastone of the adjacent sub-regions.
 8. The non-transitory computer readablemedium of claim 7, the code being further configured to program the oneor more processors to analyze a shape characteristic of the one or moresub-regions to further determine the iris region.
 9. The non-transitorycomputer readable medium of claim 7, the code being further configuredto program the one or more processors to analyze said acquired image inLuminance chrominance color space and said image comprises a region ofhigh intensity pixels having a luminance value greater than a luminancethreshold.
 10. The non-transitory computer readable medium of claim 9,wherein said region of high intensity pixels has blue-yellow chrominancevalues greater than a chrominance threshold.
 11. The non-transitorycomputer readable medium of claim 10, wherein said region of highintensity pixels has a red-green value less than a red-green threshold.12. The non-transitory computer readable medium of claim 7, the codebeing further configured to program the one or more processors toanalyze an eye-related characteristic comprising at least one of shape,roundness and pupil size to determine the iris region.
 13. A method ofsegmenting an iris region from one or more adjacent sub-regions of anidentified eye region within an acquired digital image, comprisingacquiring a digital image; identifying a candidate eye region in saidimage; determining one or more sub-regions of the candidate eye regioneach having an intensity value that is above or below one or morethreshold intensity values; identifying an iris region and one or moreadjacent sub-regions based on discerning chrominance values of pixels ofthe iris region and the one or more adjacent sub-regions; anddetermining the iris region based on analyzing one or more chrominancevalues of pixels of at least one of the adjacent sub-regions.
 14. Themethod of claim 13, further comprising analyzing a shape characteristicof the one or more sub-regions to further determine the iris region. 15.The method of claim 13, further comprising analyzing said acquired imagein Luminance chrominance color space and said image comprises a regionof high intensity pixels having a luminance value greater than aluminance threshold.
 16. The method of claim 15, wherein said region ofhigh intensity pixels has blue-yellow chrominance values greater than achrominance threshold.
 17. The method of claim 16, wherein said regionof high intensity pixels has a red-green value less than a red-greenthreshold.
 18. The method of claim 13, further comprising analyzing aneye-related characteristic comprising at least one of shape, roundnessand pupil size.